{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Description and Preprocessing "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modular "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data has been collected using the modular rig tester that shown in figure (a). The tester consists of several components : (1) electric motor; (2)torque-measurement shaft ; (3) a rolling bearing test module; (4) fly wheel; (5) load motor. A detailed explanation of the modular tester can be found [KAt].  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Description "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this dataset, rolling bearing elements with about 32 bearing experiment has been conducted as shown: \n",
    "- 6 bearings are undamaged (heakthy)\n",
    "- 12 bearings with artificial damages- \n",
    "- 14 bearings with real accelerated damages. \n",
    "\n",
    "The artificial and real damages have two fault types: inner and outer ring faults.  20 measurements are conducted for each bearing file, each file collected under 64 KHz sampling rate and last for 4 seconds. Hence, the data file can have around 256000 data points.  The experiments carried out under 4 different working conditions.  Different parameters among working conditions such as: rotational speed, load torque, and radial force. Rotational speed varied between 900 rpm and 1500 rpm, while load toque changed from 0.1 Nm to 0.7 Nm, while the asserted radial force has been increased from 400N to 1000N. Table_x shows the set up of the 4 different operating conditions. \n",
    "\n",
    "Summary: \n",
    "- The dataset has 26 damaged bearing states (i.e., 12 artificial damages and 14 real damages) and 6 healthy(undamaged) bearing states.\n",
    "- Four different operating conditions. \n",
    "- 20 measurements of 4 second each for each bearing file, where each file name has code of to represent its operating working condition.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Loading and Preprocessing "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Loading of 20 measurements for each real damage file "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this dataset we employed files with real damages damages for more practical scenario. the experiments conducted under 4 differnt working conditions. The following table shows the selected real files train our model for each wokring condition. Each of these files (e.g. KA01, KA02, KI01,....) has 20 measurements (e.g. KA01_1,.....,KA01_20). Signle measurement has 256000 data points (i.e. 64 KHz and 4 seconds long). To load these files automatically we implemented matalb code \"automatic file loading\". The output of matlab code will be (A_5120L, B5120L, C5120L, D_5120L)\n",
    "In our experiement we  used single healthy file,  5 files with outer faults, and 5 files with inner faults. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Data augmentation using sliding window "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To handle the lengthy data files with 2560000, we used sliding window to extract a short samples and applied overlapping to increase the number of samples. For each measurement we used window length of 5120 and shifiting size of 4096 as shown in the following figure. Note that for the healthy samples the shfting size reduced to 1024 to balance the number of healthy data. Finally we woll have 4900 healthy samples, 6200 inner faulty, and 6200 outer faulty samples. The following figure show the sliding window for healthy and faulty data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " The following equations show the details about calcuating number of generated samples:\n",
    "\n",
    "$$\\begin{align}\n",
    "\\textbf{n}&=ceil(\\frac{T - L}{S}) * number\\ of \\ measuerements  \\\\\n",
    "\\textbf{N}&=\\textbf{n}*number\\ of\\ class \\ files \\\\ \n",
    "\\textbf{K}&=\\textbf{N}*number\\ of\\ classes.\n",
    "\\end{align}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where $n$ is number samples per file,$N$ is the number of samples for each class, and __K__ is the total number of samples for each working conditions  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Import Libraries "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "import numpy as np \n",
    "import mat4py as sp\n",
    "from sklearn.model_selection import train_test_split\n",
    "import os "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data loading "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = 'data_files'\n",
    "output_dir = '../../data/pFD'\n",
    "# Loading data \n",
    "data_a = sp.loadmat(os.path.join(data_path, 'A_5120L.mat'))\n",
    "data_b = sp.loadmat(os.path.join(data_path, 'B_5120L.mat'))\n",
    "data_c = sp.loadmat(os.path.join(data_path, 'C_5120L.mat'))\n",
    "data_d = sp.loadmat(os.path.join(data_path, 'D_5120L.mat'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Constructing samples for different working conditions "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Construct data and labels working condition A "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "healthy_files= ['K001']; \n",
    "inner_fault_files = ['KI14','KI21','KI17','KI18','KI16']; \n",
    "out_fault_files = ['KA22','KA04','KA15','KA30','KA16'];\n",
    "\n",
    "Kat_data = {'a':{'samples':[], 'labels':[]}, \n",
    "            'b':{'samples':[], 'labels':[]},\n",
    "            'c':{'samples':[], 'labels':[]},\n",
    "            'd':{'samples':[], 'labels':[]}}\n",
    "\n",
    "# loop through healthy data \n",
    "for data_item in healthy_files:\n",
    "    wk_a_samples = torch.from_numpy(np.asarray(data_a['A_5120L']['real'][data_item]))\n",
    "    wk_b_samples = torch.from_numpy(np.asarray(data_b['B_5120L']['real'][data_item]))\n",
    "    wk_c_samples = torch.from_numpy(np.asarray(data_c['C_5120L']['real'][data_item]))\n",
    "    wk_d_samples = torch.from_numpy(np.asarray(data_d['D_5120L']['real'][data_item]))\n",
    "\n",
    "    wk_a_labels  = torch.LongTensor(wk_a_samples.size(0)).fill_(0)                                \n",
    "    wk_b_labels  = torch.LongTensor(wk_b_samples.size(0)).fill_(0)                                       \n",
    "    wk_c_labels  = torch.LongTensor(wk_c_samples.size(0)).fill_(0)                                      \n",
    "    wk_d_labels  = torch.LongTensor(wk_d_samples.size(0)).fill_(0)\n",
    "\n",
    "    Kat_data['a']['samples'].append(wk_a_samples)\n",
    "    Kat_data['b']['samples'].append(wk_b_samples)\n",
    "    Kat_data['c']['samples'].append(wk_c_samples)\n",
    "    Kat_data['d']['samples'].append(wk_d_samples)\n",
    "                                               \n",
    "    Kat_data['a']['labels'].append(wk_a_labels)\n",
    "    Kat_data['b']['labels'].append(wk_b_labels)\n",
    "    Kat_data['c']['labels'].append(wk_c_labels)\n",
    "    Kat_data['d']['labels'].append(wk_d_labels)                                            \n",
    "\n",
    "# Loop through outer faults \n",
    "for data_item in out_fault_files:\n",
    "    wk_a_samples = torch.from_numpy(np.asarray(data_a['A_5120L']['real'][data_item]))\n",
    "    wk_b_samples = torch.from_numpy(np.asarray(data_b['B_5120L']['real'][data_item]))\n",
    "    wk_c_samples = torch.from_numpy(np.asarray(data_c['C_5120L']['real'][data_item]))\n",
    "    wk_d_samples = torch.from_numpy(np.asarray(data_d['D_5120L']['real'][data_item]))\n",
    "\n",
    "    wk_a_labels  = torch.LongTensor(wk_a_samples.size(0)).fill_(1)                                \n",
    "    wk_b_labels  = torch.LongTensor(wk_b_samples.size(0)).fill_(1)                                       \n",
    "    wk_c_labels  = torch.LongTensor(wk_c_samples.size(0)).fill_(1)                                      \n",
    "    wk_d_labels  = torch.LongTensor(wk_d_samples.size(0)).fill_(1)\n",
    "                                               \n",
    "    Kat_data['a']['samples'].append(wk_a_samples)\n",
    "    Kat_data['b']['samples'].append(wk_b_samples)\n",
    "    Kat_data['c']['samples'].append(wk_c_samples)\n",
    "    Kat_data['d']['samples'].append(wk_d_samples)\n",
    "                                               \n",
    "    Kat_data['a']['labels'].append(wk_a_labels)\n",
    "    Kat_data['b']['labels'].append(wk_b_labels)\n",
    "    Kat_data['c']['labels'].append(wk_c_labels)\n",
    "    Kat_data['d']['labels'].append(wk_d_labels)        \n",
    "    \n",
    "# loop through inner faults \n",
    "for data_item in inner_fault_files:\n",
    "    wk_a_samples = torch.from_numpy(np.asarray(data_a['A_5120L']['real'][data_item]))\n",
    "    wk_b_samples = torch.from_numpy(np.asarray(data_b['B_5120L']['real'][data_item]))\n",
    "    wk_c_samples = torch.from_numpy(np.asarray(data_c['C_5120L']['real'][data_item]))\n",
    "    wk_d_samples = torch.from_numpy(np.asarray(data_d['D_5120L']['real'][data_item]))\n",
    "\n",
    "    wk_a_labels  = torch.LongTensor(wk_a_samples.size(0)).fill_(2)                                \n",
    "    wk_b_labels  = torch.LongTensor(wk_b_samples.size(0)).fill_(2)                                       \n",
    "    wk_c_labels  = torch.LongTensor(wk_c_samples.size(0)).fill_(2)                                      \n",
    "    wk_d_labels  = torch.LongTensor(wk_d_samples.size(0)).fill_(2)\n",
    "                                               \n",
    "    Kat_data['a']['samples'].append(wk_a_samples)\n",
    "    Kat_data['b']['samples'].append(wk_b_samples)\n",
    "    Kat_data['c']['samples'].append(wk_c_samples)\n",
    "    Kat_data['d']['samples'].append(wk_d_samples)\n",
    "                                               \n",
    "    Kat_data['a']['labels'].append(wk_a_labels)\n",
    "    Kat_data['b']['labels'].append(wk_b_labels)\n",
    "    Kat_data['c']['labels'].append(wk_c_labels)\n",
    "    Kat_data['d']['labels'].append(wk_d_labels)       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data Splitting\n",
    "full_data_kat={}\n",
    "for work_env in ['a', 'b', 'c', 'd']:\n",
    "    data = torch.cat(Kat_data[work_env]['samples']).numpy()\n",
    "    labels =  torch.cat(Kat_data[work_env]['labels']).numpy()\n",
    "    \n",
    "    X_train, X_test, y_train, y_test = train_test_split(data,  labels,  stratify=labels,  test_size=0.2, random_state=1)\n",
    "    X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.25, stratify=y_train, random_state=1)\n",
    "    \n",
    "    full_data_kat[work_env]= {'train':{'samples': torch.from_numpy(X_train), 'labels':torch.LongTensor(y_train)},\n",
    "                           'val':{'samples':torch.from_numpy(X_val), 'labels':torch.LongTensor(y_val)},\n",
    "                           'test':{'samples':torch.from_numpy(X_test), 'labels':torch.LongTensor(y_test)}}\n",
    "\n",
    "# datasaving\n",
    "for data_idx in ['a', 'b', 'c', 'd']:\n",
    "    torch.save(full_data_kat[data_idx]['train'], os.path.join(output_dir ,f'train_{data_idx}.pt'))\n",
    "    torch.save(full_data_kat[data_idx]['val'],  os.path.join(output_dir , f'val_{data_idx}.pt' ))\n",
    "    torch.save(full_data_kat[data_idx]['test'] , os.path.join(output_dir , f'test_{data_idx}.pt'))\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}